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EXECUTIVE SUMMARY 



In 2001, AEL was re-funded by the U.S. Department of Education to serve as the 
Regional Educational Laboratory for Kentucky, Tennessee, Virginia, and West Virginia. One 
emphasis of the contract is “School/Community Connections.” One strategy encompassed by 
the funded scope of work is partnering with faith-based organizations in Kanawha County, West 
Virginia, to (1) build their capacity to better serve at-risk youth and (2) document what best helps 
such organizations to successfully impact youth and their families. 

In 2001, AEL’s school/community work focused on helping to form a coalition of 
African American faith-based organizations in Charleston, called the Partnership of African 
American Churches (PAAC), that could collaborate to bring about systemic impact on the 
education of at-risk youth. However, after nearly a year of extensive support from AEL, it 
became apparent that community issues beyond those in AEL’s purview (such as senior citizen 
health) and insufficient ownership by PAAC members were hindering this group from focusing 
on supporting at-risk youth. Hence, the focus of AEL’s faith-based strategy changed in 2002 
from working collectively to working individually with grassroots organizations with youth 
programs focused on improving the academic achievement of high-risk and at-risk youth. 

One of the first faith-based organizations with which AEL became involved on an 
individual basis was the Helping Others Pursue Excellence (HOPE) Youth Development 
Movement. AEL project staff have been involved with the HOPE program during 2002 by 
providing external facilitation and technical assistance, such as refining record-keeping systems; 
developing intake and tracking systems; establishing linkages to other agencies and services; 
facilitating meetings; developing materials; and providing training, capacity building, data 
collection, evaluation services, program design, and research on best practices. 

Work undertaken with the HOPE program during 2002 has served as a pilot test of the 
t)^3es of data collection methods and instruments needed to document the impact of AEL’s 
involvement with faith-based organizations serving African American youth. This evaluation 
report of the process for partnering with a faith-based organization serves as documentation of 
the pilot test of the instruments and provides the baseline data for the HOPE program. The 
primary audiences for this report include AEL staff, U.S. Department of Education staff, and 
HOPE staff Secondary audiences include staff of other faith-based organizations interested in 
promoting such partnerships within their communities. 

Description of HOPE 

The HOPE Community Development Corporation began as a grassroots organization in 
Charleston, West Virginia, in 1994, and was formally established as a nonprofit faith-based 
organization in 1997. HOPE is a holistic, comprehensive, and integrated youth development 
system with linkages and support from Kanawha County schools, Kanawha County Juvenile and 
Criminal Justice systems, higher education, and community and faith-based organizations. The 
program operates in 8 high schools, 13 middle schools, and 4 elementary schools, serving more 
than 400 youth. HOPE currently operates a number of programs ranging from tutoring and 
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mentoring to youth health initiatives and career planning. It also operates as the service provider 
for the Kanawha County in-school youth component for the federal Workforce Investment Act 
(WIA) of 1998. 

Although HOPE services are comprehensive and integrated across grade levels, HOPE 
and AEL staff decided to focus collaborative work and documentation only at the high school 
level. Reasons for this included the closer relationships and access HOPE staff had with schools 
at the secondary level, the comprehensive array of services available through WIA, and the fact 
that the WIA program was just coming to finition and served as a natural starting point for an in- 
depth look at program outcomes. 

Evaluation Plan 

Three main questions frame the work currently underway in the faith-based initiative: 

(1) What types of technical assistance are needed to help build the capacity of faith-based 
organizations to design and implement programs that support and nurture the educational 
attainment of youth, especially high-risk and at-risk youth? (2) What impact does participation 
in an educational or social program run by a faith-based organization have on high-risk or at-risk 
youths’ overall educational experience (i.e., academic achievement, attendance, attitude toward 
learning, behavior)? (3) To what degree and in what ways are faith-based organizations able to 
engage family members of youth involved in their programs in activities that support the 
academic achievement of youth? 

To address these questions, AEL evaluators constructed a multiple-method evaluation 
design that addresses both process and outcomes through a case study of a faith-based 
organization. Given that AEL had worked with the HOPE program for the longest period of 
time, it was selected to serve as the case study site. The evaluation plan calls for data to be 
collected during a baseline year and each year thereafter that the project is funded at AEL. The 
baseline year for HOPE data collection was 2002. Methods include interviewing faith-based 
staff, students, and parents; gathering reflections of AEL staff; administering the AEL Skills 
Inventory survey and a Satisfaction survey to students; and compiling student record data. 

In order to comply fully with regulations pertaining to protection of human subjects, a 
signed Informed Consent Form was secured from a parent of each student who participated in 
any interview or survey data collection activities. Further, a signed memo of authorization from 
the HOPE administrator was secured before any student record data were provided to AEL. 

Data Collection 

AEL staff conducted a group interview with all six HOPE staff members in October 
2002, an individual interview with the HOPE administrator in October 2002, two group 
interviews with a total of eight students in June and July 2002, and seven individual parent 
interviews in November 2002. In December 2002, the three AEL project staff provided written 
narratives of their involvement with the faith-based organization, including their perceptions of 
successes and failures, lessons learned, and issues for future consideration. During the fall of 
2002, HOPE staff provided AEL staff with student record data for the 2001-2002 school year. 
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These data included student identification numbers, letter grades for subjects by nine-week 
periods, gender and ethnicity information, and (in some cases) attendance information for 1 74 
students. Data were merged into a SPSS file, which was used for all subsequent analyses. 
Analyses focus on within-semester grade differences and differences by gender and race. 

AEL staff provided photocopies of the AEL Skills Inventory (a self-report assessment 
tool) to HOPE staff, who distributed the survey to participating youth during the summer of 
2002. A total of 174 surveys out of approximately 300 were completed and returned to AEL 
staff in the fall of 2002 (about 58% response rate). These survey data were scanned into files 
that were then exported to SPSS for statistical analysis. The Satisfaction survey was 
administered to HOPE participants twice — once in the summer of 2002 and again in November 
2002. For the first administration, the survey was photocopied as the fourth page of the AEL 
Skills Inventory; a total of 174 completed surveys were returned out of the 300 (58% response 
rate). For the second administration, HOPE staff distributed the single-page survey in November 
2002; at this time, the total number distributed is unknown. A total of 1 19 completed surveys 
were returned to AEL staff These survey data were scanned into files that were then exported to 
SPSS for statistical analysis. 

Findings 

Results from the pilot-test administration of the data collection instruments at the HOPE 
Youth Development Movement are presented in two summary forms in the findings section of 
this report. First, to demonstrate the full extent of the data gathered, individual summaries are 
presented by each type of instrument. Second, to frame the findings around the evaluation 
questions, a summary is presented that links findings from specific methods to each question. 

While the findings by individual instmment are infomiative, they are quite lengthy; therefore, only 
the summary by evaluation questions is provided below. 

(1) What types of technical assistance are needed to help bidld the capacity of faith-based 
organizations to design and implement programs that support and nurture the educational 
attainment of youth, especially high-risk and at-risk youth? 

Through AEL staff reflections, a variety of technical assistance strategies to help build 
the capacity of faith-based organizations were noted. Staff also noted that training in grant 
writing helps such organizations develop programs to generate income to sustain their seiwices. 
Similarly, sound evaluation practices are a key part of any successful grant, and AEL can 
provide expertise in this area. A third area of technical assistance is leadership development. 
According to staff reflections, many faith-based staff and/or volunteers are unaccustomed to 
serving in leadership roles; such assistance could help these workers become more comfortable 
in this capacity. 

Interviews with HOPE staff by AEL evaluators revealed perceptions of AEL’s technical 
assistance as high quality with “organizational intelligence” to understand the importance of 
research-based curriculum and activity, measurable goals and objectives, and accurate data 
collection. HOPE staff viewed AEL’s assistance as helpful and valuable, and deemed it to be a 
great asset. 
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To help other faith-based organizations design and implement programs, HOPE staff 
suggested that such organizations must first have a clear vision of what they desire to accomplish 
in terms of helping youth. They also noted that patience was critical, i.e., a focus on the long 
term because measurable improvements take time. Further, HOPE staff noted that AEL could 
provide assistance with record keeping, legal issues and/or paperwork, and identification of 
funding opportunities. 

Several suggestions from HOPE staff for improving AEL’s technical assistance services 
included continued learning about and understanding of grassroots community-level issues and 
how community organizations operate, and more involvement of AEL staff in school-site 
activities. 

(2) What impact does participation in an educational or social program run by a faith-based 
organization have on high-risk or at-risk youths ' overall educational experience (i.e., academic 
achievement, attendance, attitude toward learning, behavior)? 

Student interviewees told AEL evaluators of HOPE’S positive impact on their academic 
progress and their self-esteem, but inspection of academic records did not support these 
perceptions. Students noted the individualized emphasis they received, and their perceptions that 
HOPE staff were truly interested in helping each child. In fact, the one topic most frequently 
mentioned as most liked about HOPE was the caring, committed staff 

On average, HOPE students had grades of B or C in each of the four subjects studied, 
with higher grades in math than in English, history, or science. Students’ math and science 
grades dropped significantly from the first to second nine-week period, yet effect sizes were 
minimal. And, while grades did drop from the first to second nine weeks, they were still well 
within the passing range. However, Black males consistently fared worse grade-wise for all four 
subjects for both grading periods. 

Students’ scores on the self-report AEL Skills Inventory ranged for the most part from 
3.5 to 4.0 (maximum score = 5) for each of seven behavioral areas (confidence, written 
communications, oral communications, values, decision making, problem solving, and planning), 
indicating that they felt mostly positive about actions pertaining to each area. 

In terms of satisfaction with HOPE services and the impact the program had on their lives, 
about three fourths of the students completing the Satisfaction survey indicated they were satisfied 
with the extent to which the HOPE services met their needs and positively impacted their lives. 

About two thirds indicated they were satisfied with new skills or knowledge they had gained and 
their spiritual growth. 

(3) To what degree and in what ways are faith-based organizations able to engage family 
members of youth involved in their programs in activities that support the academic achievement 
of youth? 

At this point in time, HOPE staff believed that family involvement in their program was 
less than satisfactory, based on interviews conducted by AEL evaluators. HOPE staff 
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interviewees mentioned that multiple siblings often participated in the program, but that parent 
involvement was lacking. They noted that activities were open to all family members, yet felt 
that most parents turned to HOPE only in a crisis situation. Further, HOPE staff also reported 
that even though parents seemed to support the program, they didn’t seem to fully understand the 
comprehensive array of services. 

Student data corroborated staffs perceptions, with few students noting active parent 
participation, according to student interviews conducted by AEL evaluators. Students thought 
parents supported their participation passively (i.e., through encouragement for attending) rather 
than actively (i.e., by becoming involved and attending activities). Finally, in interviews 
conducted by AEL evaluators, parents themselves indicated almost no active involvement in 
HOPE activities and few indicated that they actually understood the program fully. 

Conclusions 

Several conclusions can be drawn from the findings of this evaluation. 

• The data collection instruments seem to provide a comprehensive picture of a faith-based 
organization’s youth program. The mixed-method approach of surveys, interviews, 
reflections, and academic records provides a triangulation of data that compensates for 
any inherent weaknesses in a particular collection strategy. 

• The various collection methods do not seem overly burdensome to faith-based staff or 
overly intrusive to participants. Data were collected in a timely manner, once AEL and 
HOPE staff were able to coordinate their schedules. However, there were difficulties 
encountered in identifying sufficient numbers of students and parents willing to be 
interviewed. 

• One area that seems to have been left untapped is the perceptions of faith-based staff in 
terms of their communication and collaboration with their community schools. The 
interview protocols did not include any mention of this aspect. 

• It seems that the time is ripe, with the conclusion of this report, to work closely with 
HOPE staff in building their capacity for evaluative activities, interpretations, and 
utilization. HOPE staff willingly shared all requested data and have seemed most 
appreciative of the evaluation assistance received to date. 

• It also seems apparent, given AEL staff and HOPE staff input, that AEL’s future work 
with HOPE staff could focus on capacity building in the areas of leadership development 
and training in grant writing so that HOPE staff can sustain current services. 

• On average, HOPE students earned Cs in English, history, and science, and Bs in math. 
However, students’ math and science grades dropped significantly from the first to 
second nine-week period. Further, Black males consistently fared worse grade-wise for 
all four subjects for both grading periods. 
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• HOPE students report feeling positive about their actions and attitudes related to 
confidence, written communications, oral communications, values, decision making, 
problem solving, and planning. 

• HOPE students report that the HOPE services are adequately meeting their needs and 
positively impacting their lives. They show less satisfaction with knowledge or skills 
gained from their participation in the HOPE program or spiritual growth. 

• There seems to be a discrepancy regarding academic improvement as a result of HOPE 
participation. Two thirds of the students indicated they were satisfied with their 
academic improvements and interviewees noted the academic benefits of such 
participation, yet students’ grades did decline in four subject areas across grading periods. 

Recommendations 

Given the conclusions noted previously, several recommendations are suggested for 
improving any future data collection efforts. 

• AEL project and evaluation staff need to work closely with faith-based staff to strategize 
ways to promote more participation (number and depth) of both students and parents in 
future group interviews. Perhaps one such strategy might be to schedule an evening 
interview, with dinner provided, so that parents and students could attend and participate 
without feeling pressured to skip or rush through the interview. 

• AEL could consider tailoring future technical assistance to HOPE staff toward training in 
grant writing and leadership, if HOPE staff are receptive to this suggestion and willing to 
work on these endeavors. 

• The interview protocol for faith-based staff should be modified to include questions 
pertaining to their perceptions of school communication/collaboration. 

• HOPE staff may want to investigate possible causes for students’ grade declines across 
grading periods. And, they may consider monitoring students’ grades more closely by 
grading periods, to try to intervene as necessary to avoid such declines. Further, they 
may want to intensify their tutoring activities with Black males, given their consistently 
lower grades. 
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INTRODUCTION 



In 2001, AEL was re-funded by the U.S. Department of Education to serve as the 
Regional Educational Laboratory for Kentucky, Tennessee, Virginia, and West Virginia. One 
emphasis of the contract is “School/Community Connections.” While communities and schools 
have a common interest in helping children develop into productive citizens, few schools and 
communities work collaboratively on behalf of children, particularly African American children 
(AEL, 2000; Kusimo, 1999). 

One strategy encompassed by the funded scope of work is partnering with faith-based 
organizations in Kanawha County, West Virginia, to (1) build their capacity to better serve at- 
risk youth and (2) document what best helps such organizations to successfully impact youth and 
their families. 

In 2001, AEL’s school/community work focused on helping to fonn a coalition of 
African American faith-based organizations in Charleston that could collaborate to bring about 
systemic impact on the education of at-risk youth. AEL staff did the preliminary planning and 
introductions, convened the group, provided meeting space, facilitated meetings, and provided 
technical assistance to support the group. Called the Partnership of African American Churches 
(PAAC), this group formed a nonprofit corporation, named officers, and met throughout the year 
as it endeavored to coalesce into a more powerful force than members would have been 
individually. 

However, after nearly a year of extensive support from AEL, it became apparent that 
community issues beyond those in AEL’s purview (such as senior citizen health) and insufficient 
ownership by PAAC members were hindering this group from focusing on supporting at-risk 
youth. Hence, the focus of AEL’s faith-based strategy changed in 2002 from working 
collectively to working individually with grassroots organizations with youth programs focused 
on improving the academic achievement of high-risk and at-risk youth. AEL staff used Schorr’s 
(1998) “Seven Attributes of Highly Effective Programs” as a framework for identifying specific 
faith-based organizations with which to collaborate. According to Schorr, successful programs 
(1) are comprehensive, flexible, responsive, and persevering; (2) see children in the context of 
their families; (3) deal with families as parts of neighborhoods and communities; (4) have a long- 
term, preventive orientation, a clear mission, and continue to evolve over time; (5) are well 
managed by competent and committed individuals with clearly identifiable skills; (6) have staff 
that are trained and supported to provide high-quality, responsive services; and (7) operate in 
settings that encourage practitioners to build strong relationships based on mutual tmst and 
respect. 



One of the first faith-based organizations with which AEL became involved on an 
individual basis was the Helping Others Pursue Excellence (HOPE) Youth Development 
Movement. AEL project staff have been involved with the HOPE program during 2002 by 
providing external facilitation and technical assistance, such as refining record-keeping systems; 
developing intake and tracking systems; establishing linkages to other agencies and services; 
facilitating meetings; developing materials; and providing training, capacity building, data 




13 



2 



collection, evaluation services, program design, and research on best practices. AEL staff 
subsequently began collaborations with seven other faith-based organizations: Covenant 
Christian Fellowship Church, First Baptist Church, First Presbyterian Church, Kanawha Institute 
for Social Research and Action, New Covenant Missionary Baptist Church, Rimson Memorial 
Church of God in Christ, and Simpson United Methodist Church. 



Purpose of Report 



Work undertaken with the HOPE program during 2002 has served as a pilot test of the 
types of data collection methods and instruments needed to document the impact of AEL’s 
involvement with faith-based organizations serving African American youth. This evaluation 
report of the process for partnering with a faith-based organization serves as documentation of 
the pilot test of the instruments and provides the baseline data for the HOPE program. The 
primary audiences for this report include AEL staff, U.S. Department of Education staff, and 
HOPE staff. Secondary audiences include staff of other faith-based organizations interested in 
promoting such partnerships within their communities. 



HOPE Description 



The HOPE Community Development Corporation began as a grassroots organization in 
Charleston, West Virginia, in 1994, and was fomially established as a nonprofit faith-based 
organization in 1997. The HOPE Youth Development Movement is a holistic, comprehensive, 
and integrated youth development system with linkages and support from Kanawha County 
schools, the Kanawha County Juvenile and Criminal Justice systems, higher education, and 
community and faith-based organizations. Within Kanawha County, the program operates in 8 
high schools, 13 middle schools, and 4 elementary schools, serving more than 400 youth. HOPE 
currently operates a number of programs ranging from tutoring and mentoring to youth health 
initiatives and career planning. See Appendix A for a complete listing and description of cun'ent 
programs. 

HOPE also operates as the service provider for the Kanawha County in-school youth 
component for the federal Workforce Investment Act (WIA) of 1998. This act mandated that 
states develop local Workforce Investment Boards to provide education and job training services 
to adults, dislocated workers, and youth (both in-school and out-of-school) through One-Stop 
systems. Participating youth must be low-income and ages 14 to 21 (although up to 5% who are 
not low-income may receive services if they face certain barriers to school completion or 
employment). They must also face one or more of the following challenges: (1) a school 
dropout; (2) a basic literacy skills deficiency; (3) a homeless, runaway, or foster child; 

(4) pregnant or a parent; (5) an offender; or (6) in need of help completing an education 
program or securing and holding a job. 
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One service HOPE provides is a stipend that youth may earn on a nine-week basis 
(maximum of $40 per month) during the school year. This incentive is used to encourage 
students to stay in school, participate in HOPE activities, and avoid negative behaviors. 
Deductions are taken if certain criteria are not met within the grading period. Such criteria 
include no school disciplinary referrals, no suspensions, not more than three absences, 
participation in all mandatory HOPE activities and at least one optional HOPE activity per 
month, participation in at least one tutoring session per week if less than a 3.0 grade point 
average, no HOPE disciplinary referrals, and showing their report cards to HOPE staff. 

Although HOPE services are comprehensive and integrated across grade levels, HOPE 
and AEL staff decided to focus collaborative work and documentation only at the high school 
level. Reasons for this included the closer relationships and access HOPE staff had with schools 
at the secondary level, the comprehensive array of services available through WIA, and the fact 
that the WIA program was just coming to fruition and served as a natural starting point for an in- 
depth look at program outcomes. 
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METHODS 



Evaluation Questions 



Three main questions frame the work currently underway in the faith-based initiative: 

1 . What types of technical assistance are needed to help build the capacity of faith-based 
organizations to design and implement programs that support and nurture the educational 
attainment of youth, especially high-risk and at-risk youth? 

2. What impact does participation in an educational or social program run by a faith-based 
organization have on high-risk or at-risk youths’ overall educational experience (i.e., 
academic achievement, attendance, attitude toward learning, behavior)? 

3. To what degree and in what ways are faith-based organizations able to engage family 
members of youth involved in their programs in activities that support the academic 
achievement of youth? 

To address these questions, AEL evaluators constructed a multiple-method evaluation 
design that includes both process and outcomes through a case study of a faith-based 
organization. Given that AEL had worked with HOPE for the longest period of time, it was 
selected to serve as the case study site. This design includes four attributes necessary for 
evaluating complex interventions (Schorr, 1998): (1) a strong theoretical/conceptual base, (2) an 
emphasis on shared interests rather than adversarial relationships between agencies, (3) use of 
multiple methods, and (4) a rigorous and relevant design. Specific data collection strategies are 
described below for each question. The evaluation plan calls for data to be collected during a 
baseline year and each year thereafter that the project is funded at AEL. The baseline year for 
HOPE data collection was 2002. 

Methods for addressing these evaluation questions include interviewing faith-based staff, 
students, and parents; gathering reflections of AEL project staff; administering the AEL Skills 
Inventory survey and a Satisfaction survey to students; and compiling student record data, 
including achievement data. Table 1 provides a matrix that aligns the three questions with 
specific data collection strategies. These data collection procedures are described at length in the 
following Instrumentation section. 

AEL evaluators worked during 2002 to develop the draft data collection instruments, i.e., 
interview protocols and surveys. All instruments have been developed, reviewed, and pilot 
tested with the HOPE program and reviewed by AEL’s Institutional Review Board for protection 
of human subjects. Further, the evaluation has been aligned with The Program Evaluation 
Standards (Joint Committee on Standards for Educational Evaluation, 1994); see Appendix B for 
a completed checklist. 
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Table T. Matrix of Evaluation Questions and Data Collection Strategies 



Data Collection 
Strategies 


Evaluation Questions 


(1) What types of tech- 
nical assistance are 
needed to help build the 
capacity of faith-based 
organizations to design 
and implement programs 
that support and nurture 
the educational attain- 
ment of youth, especially 
high-risk and at-risk 
youth? 


(2) What impact does 
participating in an educa- 
tional or social program 
run by a faith-based 
organization have on high- 
risk or at-risk youths’ 
overall educational 
experience (i.e., academic 
achievement, attendance, 
attitude toward learning, 
behavior)? 


(3) To what degree and 
in what ways are faith- 
based organizations able 
to engage family 
members of youth 
involved in their pro- 
grams in activities that 
support the academic 
achievement of youth? 


Faith-based staff 
interviews 








Student interviews 








Parent interviews 








AEL Skills Inventory 








Satisfaction survey 








AEL project staff 
reflections 








Student records, inc. 
achievement data 









Instrumentation 



Interview protocols. Three protocols have been developed for use with faith-based staff, 
participating youth, and parents of involved youth. The staff protocol includes 10 questions that 
focus on a description of the youth program and staff involvement, main strengths and 
weaknesses of the program, differences from other programs, involvement of family members, 
suggestions for improvement, value of AEL’s involvement, and needed technical assistance for 
other faith-based organizations implementing similar programs. The student protocol includes 
10 questions that focus on how they became involved in the youth program, types of activities in 
which they participate, how the program operates, main strengths and weaknesses of the 
program, benefits gained from participation, differences from other programs, involvement of 
their family members, suggestions for improvement, and other comments. The parent protocol 
contains 8 questions that focus on how their child became involved in the youth program, how 
the program operates, types of activities, main strengths and weaknesses of the program, benefits 
gained from participation, differences from other programs, and suggestions for improvement. 
See Appendix C for copies of the interview protocols. 

AEL staff reflections on lessons learned. One key part of this documentation is the 
perceptions and reflections of the three involved AEL project staff and what they have learned to 
date. With their constant involvement in day-to-day activities and conversations, they have an 
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understanding of how they have worked with faith-based organizations to develop and/or 
implement programs that positively impact at-risk youth. For this pilot test, AEL project staff 
provided a written narrative of their involvement with the faith-based organization, including 
their perceptions of successes and failures, lessons learned, and issues for future consideration. 

Student record data. Each faith-based organization keeps student records of relevant 
school-indicator data (i.e., grades, attendance, disciplinary referrals) and data related to its 
specific program. AEL evaluators utilized that data from HOPE in this report. 

AEL Skills Inventory. This self-report survey contains 70 items that focus on skills 
youth may or may not possess. Adapted primarily from the Leadership Skills Inventory (Karnes 
& Chauvin, 1985), the survey contains seven 10-item scales; confidence, written 
communications, oral communications, values, decision making, problem solving, and planning. 
Students respond to each item by rating their feelings on a 5-point scale of Almost Never (1) to 
Almost Always (5). This instrument was used to document baseline attitudinal and behavioral 
attributes and will be re-administered yearly to measure changes. A unique identification 
number is generated on the survey, so individual student comparisons can be made in subsequent 
analyses. This survey was printed on one sheet of 1 1 x 17 perforated paper, folded in half The 
back page was devoted to the Satisfaction survey (described next). See Appendix D for a copy 
of this inventory. 

To assess the degree to which items measure the same construct (internal consistency), 
Cronbach alpha reliability estimates were computed for this set of respondent scores. At .97, the 
coefficient was deemed to be very satisfactory for this type of instrument; the coefficient was 
also .97 for both males and females. At the subscale level, the coefficients ranged from .84 to 92 
for the full group; from .81 to .91 for females; and from .82 to .93 for males. See Table 2 for 
subscale Cronbach alpha reliability coefficients for all students and by gender, as well as a listing 
of the items that make up each subscale. 

Satisfaction survey. This survey provides the faith-based organization with student 
perceptions that are based directly on services provided. It asks students to identify their school 
(for use by the faith-based organization), then to select from a list of 14 options all of the 
activities in which they participated during the past year in the youth program. Possible 
activities include character education, field trips, postsecondary education awareness, youth 
entrepreneurship training, WIA initiative, alternative sentencing, transition/re-entry, 
tutoring/mentoring, career awareness, college preparation, abstinence education, job referral, 
intervention for juvenile offenders, and truancy diversion. Students then are asked to rate their 
satisfaction with eight items, such as quality and appropriateness of services, extent to which 
needs were met, and spiritual and academic growth, using a 5-point scale of Very Dissatisfied (1) 
to Very Satisfied (5). The last two questions are open-ended and ask students to identify what 
they like most and least about the program. See Appendix E for a copy of this survey. 

To assess the internal consistency, Cronbach alpha reliability estimates were computed 
for both the summer and fall 2002 sets of respondent scores for the eight satisfaction items. For 
both administrations, the coefficients were deemed to be very satisfactory, at .93 and .92. 
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Subscale Name 


Alpha 

Coefficient 


Items 


Confidence 


.84 for 
full group 

.81 for 
females 

.87 for 
males 


1. I am entitled to my own opinion. 

2. I can state my opinion when others disagree. 

3. I can identify my weaknesses. 

4. I can accept constructive criticism. 

5. I feel comfortable in most situations. 

6. I can take a stand when the going gets rough. 

7. I like to face my problems head on. 

8. I can resist when other people impose on me. 

9. I can insist that others respect my rights. 

10. I can identify my strengths. 


Written 

Communications 


.88 for 
full group 

.91 for 
females 

.82 for 
males 


1 1 . I can write my ideas so others understand them. 

12. I can distinguish fact from opinion in writing. 

13. I can compare and contrast ideas in writing. 

14. I can summarize written information. 

15. I know how to use written information. 

1 6. I can write to persuade others to my viewpoint. 

17. I can write a speech. 

18. I can evaluate my writing. 

1 9. I can prepare an agenda for a meeting. 

20. I can write an outline. 


Oral 

Communications 


.90 for 
full group 

.89 for 
females 

.91 for 
males 


21 . I can speak in a clear manner. 

22. I can express a group’s ideas. 

23. I can orally defend my viewpoint. 

24. I can deliver a prepared speech to a group. 

25. I can moderate panel discussions. 

26. I can use effective body language as I speak. 

27. I am sincere when speaking. 

28. I can deliver a speech spontaneously. 

29. I can tell others how I feel. 

30. I can effectively listen to others. 


Values 


.91 for 
full group 

.83 for 
females 

.92 for 
males 


31. I have a set of personal standards. 

32. I recognize the important things in my life. 

33. lam loyal to those closest to me. 

34. I can accept other people’s ideas. 

35. I do what I say I will. 

36. I try to deal honestly with others. 

37. I can accept other people’s values. 

38. I try to respect the feelings of others. 

39. I understand my own feelings. 

40. I treat other people fairly. 



O 

ERIC 
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Table 2 (continued) 



Subscale Name 


Alpha 

Coefficient 


Items 


Decision Making 


.90 for 
full group 

.87 for 
females 

.91 for 
males 


41. I can gather facts for decision making. 

42. I can analyze facts before making a decision. 

43. I am aware of how my decisions affect others. 

44. I know how to reach logical conclusions. 

45. I can reach decisions on my own. 

46. I can accept that my decisions may not be popular. 

47. I can support group decisions, even if I don’t agree. 

48. I can make accurate decisions quickly. 

49. I understand decision making skills. 

50. I can accept advice from others. 


Problem Solving 


.92 for 
full group 

.91 for 
females 

.92 for 
males 


51. I know the elements of problem solving. 

52. I know what to do as a leader in solving problems. 

53. I can identify a problem. 

54. I can develop different ways to solve problems. 

55. I can select the best way to solve a problem. 

56. I can judge how effective my strategy is. 

57. I can make people feel safe expressing their ideas. 

58. I can resolve conflicts within a group. 

59. I can work effectively for a compromise. 

60. I can distinguish between influence and manipulation. 


Planning 


.91 for 
full group 

.86 for 
females 

.93 for 
males 


61. I have organizational skills. 

62. I set reachable goals for myself 

63. I seek advice when necessary. 

64. I can tell what is needed to accomplish goals. 

65. I can set realistic deadlines. 

66. I can accept change. 

67. I can meet deadlines. 

68. I can determine whether goals are completed. 

69. I am not overwhelmed by details. 

70. I can set objectives to help achieve my goals. 



Protection of human subjects. In order to comply fully with these regulations, a signed 
Informed Consent Form was secured from a parent of each student who participated in any 
interview or survey data collection activities. Further, a signed memo of authorization from the 
HOPE administrator was secured before any student record data were provided to AEL. 
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Data Collection 



Interviews. All six HOPE staff were interviewed in October 2002. An AEL evaluator 
conducted a group interview of the six staff members at the HOPE site, and then individually 
interviewed the HOPE administrator. AEL evaluators also conducted two group interviews of a 
convenience sample of high school students participating in the HOPE program. HOPE staff 
identified these students and secured their willingness to participate. The first group interview 
took place on June 25, 2002, with three participants. Given the low number of participants, a 
second group interview with students took place on July 19, 2002, with five participants. Both 
interviews took place at the HOPE site. While it was not possible to bring together a group of 
parents at one time and central location for this baseline data collection, an AEL evaluator spent 
a Saturday in November 2002 at the HOPE site as parents brought their children in to pick up 
their HOPE stipend checks. As parents entered the office, they were asked by HOPE staff if they 
would be willing to participate in a short individual interview about their perceptions of the 
HOPE program. Seven parents agreed to be interviewed. 

AEL staff reflections on lessons learned. For this evaluation, the three involved AEL 
project staff provided written narratives of their involvement with the faith-based organization, 
including their perceptions of successes and failures, lessons learned, and issues for future 
consideration. These narratives were completed in December 2002 and were then summarized 
by AEL evaluators and merged into one comprehensive section of this report. 

Student records. During the fall of 2002, HOPE staff provided AEL evaluators with 
student record data for the 2001-2002 school year in two fomiats: an Access database and a 
paper file. The database included student identification numbers, letter grades for subjects by 
nine-week periods, and (in some cases) attendance information for 1 74 students (no standardized 
achievement data were provided)! The paper file contained gender and ethnicity infonnation for 
each student by identification number. These data were merged into a SPSS data file, which was used 
for all subsequent analyses. Given the varying ages, races, and socioeconomic status of students 
participating in the HOPE program, it was not possible to establish a representative control 
group. Analyses focus on within-semester grade differences and differences by gender and race. 

AEL Skills Inventory. AEL evaluators provided photocopies of this survey to HOPE 
staff, who distributed the survey to participating youth during the summer of 2002. Out of 
approximately 300 surveys, a total of 1 74 were completed and returned to AEL in the fall of 
2002 (about 58% response rate). These survey data were scanned into files that were then 
exported to SPSS for statistical analysis. 

Satisfaction survey. This survey was administered to HOPE participants twice — once in the 
summer of 2002 and again in November 2002. For the first administration, the surv'ey was photo- 
copied as the fourth page of the AEL Skills Inventory. As noted above, a total of 174 completed 
surveys were returned. For the second administration, HOPE staff distributed the single-page survey 
to students on a Saturday in November as they arrived to pick up their stipend checks. A total of 1 19 
completed surveys were returned to AEL; at this time, the total number distributed is unknown. 

These survey data were scanned into files that were then exported to SPSS for statistical analysis. 
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Data Analyses 



Interviews. All interview data from the three groups (HOPE staff, parents, and 
participating youth) were analyzed in the same manner. Interview notes were first typed and 
edited for clarity. Then, summaries were prepared for each group of interviews (i.e., staff, 
parents, and youth) by AEL evaluators. Common themes emerging from individual responses 
were identified in each summary. Data were summarized in a narrative format. 

AEL staff reflections on lessons learned. The three involved AEL project staff were 
asked to provide quasi-structured narratives that, in general, focused on perceived successes and 
failures, lessons learned, and issues for future consideration. All three provided these narratives, 
which AEL evaluators reviewed and synthesized into one comprehensive summary. 

Student records. Students’ grades (A, B, C, D, or F) for English, math, science, and 
history were converted to a numeric scale (F = 0 through A = 4) using SPSS. Descriptive 
statistics (number, mean, standard deviation) were calculated for each subject. General Linear 
Model (GLM) repeated measures Analyses of Variance (ANOVAs) were conducted to determine 
if there were statistically significant differences between students’ first and second nine-week 
grades and by a grouping of gender and race (Black males. Black females. White males. White 
females) for each subject. The alpha level for detecting significant differences was set at a 
traditional .05 level and Student-Newman-Keuls post hoc analyses were utilized for pinpointing 
any significant differences among the four gender/race groups. The partial eta squared method 
for calculating effect sizes was generated via SPSS for any significant ANOVAs. Data were 
summarized via narrative text, tables, and figures. 

AEL Skills Inventory. Using SPSS, each set of items that comprised a subscale was 
added together and divided by the total number of items (10) to create subscale mean scores. 
Descriptive statistics (number, mean, standard deviation) were calculated for each of the seven 
subscales. One-way ANOVAs were conducted to determine if there were statistically significant 
differences by grade level or gender for each subscale. The alpha level for detecting significant 
differences was set at a traditional .05 level and Student-Newman-Keuls post hoc analyses were 
utilized for pinpointing any significant differences by grade level. Cohen’s d method was used 
to manually calculate effect sizes for any significant ANOVAs. Data were summarized via 
narrative text, tables, and figures. 

Satisfaction survey. SPSS was utilized to analyze these survey data. Response 
percentages and descriptive statistics (number, mean, standard deviation) were calculated for 
survey items, as appropriate. One-way ANOVAs were conducted to determine if there were 
statistically significant differences by administration for eight items pertaining to the HOPE 
services they had received. The alpha level for detecting significant differences was set at a 
traditional .05 level. Cohen’s d method was used to manually calculate effect sizes for any 
significant ANOVAs. Data were summarized via narrative text, tables, and figures. 
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FINDINGS 



Findings from the pilot-test administration of the data collection instruments at the HOPE 
Youth Development Movement are presented in two summary forms. First, to demonstrate the 
full extent of the data gathered, individual summaries are presented by each type of instrument. 
Second, to frame the findings around the evaluation questions, a summary is presented that links 
findings from specific methods to each question. 



Summaries by Instrument 



Faith-Based Staff Interviews 

A group interview of all six HOPE staff members plus an individual interview with the 
HOPE administrator were conducted at the HOPE office in October 2002. Staff were first asked 
to describe the HOPE program. One staff member described HOPE as a “holistic, compre- 
hensive youth development system.” Another said it was a system designed to provide 
assistance to youth in areas of academics, preparation for postsecondary education, and entry 
into the workforce. HOPE’S approach was described as a customized strategy to deal with each 
youth as an individual. Staff also noted that HOPE provides other services to low-income youth 
and families, including referrals for education and employment, tutoring, acting as an advocate 
for families with the court system, and referring families to debt management agencies. Another 
staff member said the HOPE system “empowers, educates, encourages, and edifies youth and 
their lives via school, church, and education programs.” Other staff see HOPE as a mediator for 
dysfunctional families and as a model for the community of how parents can get involved in their 
children’s lives. 

When asked to describe their involvement with HOPE, several staff members mentioned 
that it was a great opportunity to work with the youth in the community. Staff reported feeling 
encouraged by the changes they witness in the youth and their families and that they are making 
a difference in the lives of the youth and are energized and motivated to continue the work. One 
staff member noted that some changes take time but others are instant, such as providing jobs 
and food to needy families. Staff described HOPE as trying to build relationships with people so 
they have an ally in their comer. Staff members have been involved with HOPE for various 
lengths of time, from less than two years to as long as five years. One staff member began as a 
volunteer and is now employed by HOPE. 

Staff members were asked to identify the main strengths of the HOPE program. The 
importance of dedicated and committed staff was a recurring theme. Staff reported coming 
together in the spirit of collaboration and cooperation for the kids. Staff members see youth as 
resources with untapped potential. To them, the belief that youth can develop in school and 
come to be seen as contributing members of society is a key concept. Other staff members 
reported feeling compelled to work with youth because of a dedication and love for kids and feel 
their work is not a job but an outgrowth of serving the Lord. 
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Staff members were then asked to identify the main weaknesses of the HOPE program. 
Overall, staff reported feeling overwhelmed by the volume of work due to trying to help as many 
families as possible. Limited finances, a shortage of staff, youth transportation, and internal and 
external communication issues also were identified as weaknesses. 

When asked how the HOPE program differed from other types of tutoring/mentoring 
programs, staff members’ responses varied. Differences cited were the fact that HOPE is faith- 
based, HOPE serves the whole family and not just the child, and HOPE looks at each youth as an 
individual. One staff member didn’t see it as being different but added, “HOPE is more involved 
as far as trying to meet the needs of youth.” 

HOPE staff were asked to describe the involvement of family members of youth 
participating in HOPE activities. Staff members indicated that involvement from siblings is 
present but parent involvement is lacking. Activities are open to all family members and parents 
are strongly encouraged to attend. HOPE is currently working on setting up a parent advisory 
council that will meet bimonthly. Staff members reported that most families do not fully 
understand the program and that many families contacted HOPE only when a crisis arose. 

When asked for suggestions for improving the HOPE program, increased funding was 
suggested most often. Staff reported that increased funding would allow additional staff to be 
hired, provide for additional resources such as Hi-Y and cultural events, and increase fringe 
benefits for staff. Other suggestions included more parent and youth involvement in non- 
mandated activities, continued assessment of the effectiveness of the program, cell phones for 
staff, more staff meetings to stay focused, and getting outside involvement with the program. 

Staff members were asked what value they place on HOPE’S involvement with AEL in 
impacting at-risk youth. They cited technical assistance with survey development and data 
collection, and an understanding of the importance of measurable goals and objectives, as great 
assets of the involvement with AEL. One staff member said that AEL offers “elevated 
organizational intelligence to understand the importance of research-based curriculum and 
activity.” 

When asked what AEL could do to improve its services in working with HOPE, staff 
suggested that AEL should continue to learn about grassroot community-level issues and how 
community organizations operate. Another suggestion was that AEL should work with HOPE 
staff on a regular basis in participating schools. Staff also were interested in learning more about 
AEL services in order to know how those services could be applied to the HOPE program. 

Finally, HOPE staff were asked what type of technical assistance is needed for other 
faith-based organizations to design and implement programs that support the educational 
attainment of at-risk youth. Staff members reported that first you must have a clear vision of 
how the program will help youth and an understanding that it may not be automatically effective. 
Organizations must be in it for the duration because there are no quick returns for seeing 
measurable improvement. Other comments included database management assistance for record 
keeping, understanding legal issues and paperwork, knowledge of funding opportunities, and 
committed staff. 
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Student Interviews 

AEL conducted two group interviews with a convenience sample of high school students 
participating in the HOPE Youth Development Movement. HOPE staff identified these students 
and secured their willingness to participate. The first group interview was held on Tuesday, June 
25, 2002, with three students. The second was held on Friday, July 19, 2002, with five students. 
Both group interviews took place at Grace Bible Church, where the HOPE program was located 
at that time. 

The students participating in the group interviews had been involved with the HOPE 
program for various lengths of time. One student had been involved for only four weeks, 1 for 
four months, 2 for approximately a year, 2 for two years, and another student had participated for 
four years. One student did not comment. 

The first question asked students to describe the types of HOPE activities in which they 
had participated. Summer jobs were mentioned by several of the students. The students 
mentioned that they were paid $5.15 per hour for their work. Students also reported participating 
in community service projects such as painting houses and general duties. These community 
service projects were not mandatory activities and were usually scheduled about once a month. 
Tutoring sessions were also available for students once or twice a week in all subject areas. 
Students noted that tutoring was a requirement if they received below a 2.0 grade point average 
[Note: Tutoring is a requirement by HOPE if grade point average is below a 3.0]. Students also 
participated in other activities such as basketball games and reward trips. 

Students were asked how they got involved with HOPE; again, responses varied. Two 
students got involved because their friends were involved. One student said, “My school pretty 
much mandated that I attend.” One student was encouraged to join by his mother. Other 
students reported learning of the program from the school counselor, the employment office, and 
advertisements at their school. 

When asked to describe how the HOPE program operates, students indicated that all 
participants must first take the Test of Adult Basic Education (TABE®), which measures basic 
reading, mathematics, and language skills. Students noted they then must complete required 
paperwork before participating in program activities and that they can earn up to $120 each 
quarter by regularly attending school, staying out of trouble, attending mandatory activities, and 
keeping their grades up. Deductions may result due to lack of participation, failing grades, and 
behavioral problems. Students also noted that there were rewards for doing their work, such as a 
trip to Six Flags amusement park. 

Several students also described the HOPE program as a place to find help. Students 
reported that HOPE staff are always available to talk, to help them find a job after graduation, 
and to help with school. 

Academics and self-esteem seem to be the main strengths of the HOPE program. Several 
students said the program helps with school and their grades. One student said, “It helps you 
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carry yourself better.” Other reported strengths included the HOPE staff and the money (a 
reference to the stipend). 

Students were then asked about the benefits they had gained from participating in the 
HOPE program. Students again reported that HOPE is a place to find help. One student added, 
“[HOPE] helps you get your life straight if you have problems.” Other benefits mentioned were 
it gives students something to do, helps students set future goals, and the activities available. 

The main weaknesses of the HOPE program cited by students were more diverse than the 
strengths. Students replied that tutoring should occur more often and during the school day 
because not all students can stay after school. Other noted weaknesses were scheduled times of 
meetings, not enough advance notice of scheduled activities, and not enough field trips. 

When asked how the HOPE program differed from other types of tutoring/mentoring 
programs, only two of the students had participated in other programs. One student responded, 
“HOPE is better because there are more people and they try to help you better yourself for life 
not just with school.” This student had participated in activities at local community centers. 
Another student had been involved with Girl Scouts and said there were more group activities 
with HOPE. 

Students were asked to describe the involvement of their family members in HOPE 
activities. Four students said their mothers had participated in program activities, encouraged 
them to be involved, or were interested in working with the HOPE program. One student said 
her cousins were also involved in the HOPE program and another student reported no family 
involvement. 

Students were asked for suggestions that would improve the HOPE program. All of the 
responses were related to scheduling. Comments included activities and meetings should be 
scheduled for later in the morning and that students need more advance notification of activities. 
Overall, students suggested there should be better coordination and scheduling of HOPE 
activities. 

Finally, students were given the opportunity to make any other comments regarding their 
participation in the HOPE program. While most of the students said they had nothing further to 
add, three students made additional comments. One student again referred to the money that 
could be earned by participating in the program. One student said, “You are here because you 
want to be here” and another added, “It is not just a place to hang out with friends.” 



Parent Interviews 

While it was not possible to bring together a group of parents at one time and central 
location, an AEL evaluator spent a Saturday in November at the HOPE office as parents brought 
their children in to pick up their HOPE stipend checks. As parents entered the office, they were 
asked if they would be willing to participate in a short interview about their perceptions of the 
HOPE program. A total of seven parents agreed to be interviewed. 
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Parents were first asked how their children got involved with HOPE. Several parents 
said their children got involved through school, one indicated the job placement office, and one 
parent didn’t know. . When asked how the HOPE program operated, parents most frequently 
mentioned their children receive a check for regularly attending school. Other comments 
included HOPE provides summer jobs, opportunities for trips, and is very good for kids. One 
parent was not familiar with how the program operated. 

Parents were asked to describe the HOPE activities in which they or their children 
participated. Parents noted that several of the children were currently participating in the job 
referral program and also were receiving tutoring. Three parents indicated they had attended an 
informational meeting at the Marriott Hotel the day before being interviewed. Two parents said 
they had not attended any HOPE activities. 

The main strengths of the HOPE program that parents mentioned were that it helped their 
children to stay involved in school, provided job opportunities, and offered money for attending 
school. One parent was unclear about the strengths of the program but indicated a desire to 
become more involved with HOPE. 

Parents were then asked what benefits their children or families had gained from 
participating in the HOPE program. Parents seem most willing to talk about the benefits their 
children had received. Parents mentioned tutoring, increased self-esteem, a sense of belonging, 
and recreational activities as the greatest benefits for their children. One parent said, “[HOPE] is 
for all kids.” It provides music, food, and a time for sharing ideas and philosophies. Two 
parents also indicated that their children benefited from the summer jobs and the money 
(referring to the stipend). 

Parents also were given the opportunity to identify any weaknesses of the HOPE 
program. However, none of the parents who were interviewed could identify any weaknesses. 

Parents were asked how the HOPE program differed from other types of youth 
development programs. The parents interviewed had no prior experience with other programs. 
Finally, parents were asked what suggestions they would make for improving the HOPE 
program. One parent noted that advance notification for activities or phone calls would be good; 
the remaining parents had no suggestions for improvement. 



Staff Reflections on Lessons Learned 

The three involved AEL project staff provided reflective comments related to their 
involvement with the HOPE Youth Development Movement and the lessons they have learned to 
date. These comments reflect on perceived successes and failures, lessons learned, and issues for 
future consideration. These may be helpful in identifying effective strategies for fostering 
development of youth, which is the goal of documenting what works in effective school- 
community partnerships under AEL’s School/Community Connections thrust. A summary of 
these narratives is provided below. 
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One crucial element of HOPE’S success is the zeal and will the staff of HOPE bring to 
their work. They are genuinely caring people who see their work as part of a mission from God, 
and this is one of the strengths of their work. The rapport that they have with youth, many of 
whom would be considered throwaways by society, is great. The staff never give up and 
continue to press for the best for the youth they serve. 

The informal knowledge that HOPE staff have about their program participants is key to 
helping them serve their community. They know things about the youths’ backgrounds, home 
lives, community, social interactions, and so on, that would be impossible for someone who had 
a more distant relationship to know. 

One downside of working at HOPE is the modest compensation for the long hours 
program staff work. Because staff do so much for so little, the tendency is actually a 
de-motivator to write grants that realistically cover the costs of programs. In the long run, this 
may work against the quality of services the program can provide, because when better 
opportunities present themselves staff may be enticed to take better-paying jobs with benefits. 
Too, program staff may work more than one job to make ends meet, increasing fatigue and 
draining talent and energy away from HOPE work. 

There also is a tendency to try and provide services to anyone that wants to participate 
rather than limiting program size to the number of people the program is designed to 
accommodate. A related problem is trying to be all things to all people. With limited staff and 
budget it is almost impossible to offer high-quality services in all areas, e.g., tutoring, mentoring, 
sex education, and so on, to all age groups. It may be more effective for HOPE to narrow its 
focus to middle and high school students only, already the majority of their participants. 

Two major needs of HOPE are funding and technical assistance with building their 
capacity for grant writing. If AEL could help HOPE staff develop programs that would generate 
income to sustain their programs, that would be the way to go. It is difficult for organizations 
such as HOPE to plan to implement long-term initiatives because their sources of funding vary 
so much from year to year. 

Although these programs have begun to recognize how important evaluation of their 
work is, they do not allocate funding for this item. We are moving into a time when potential 
funders will require evidence and AEL has not made an effort to develop their expertise, even at 
a modest level, in this area. 

Through the original work with the Partnership of African American Churches, and in the 
current work with the HOPE Youth Development Movement, it has been evident that time for 
building commitment and trust is essential. 

One factor needed for successful interaction with a faith-based organization is a “cultural 
bridge” — someone who understands both AEL’s work and the faith community environment. 

The person serving as this bridge needs to be viewed credibly by the faith community. AEL’s 
involvement with faith-based organizations relies heavily on personal commitment to initiate and 
sustain ongoing relationships. 
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In initiating work with a faith-based organization, AEL needs to clearly articulate the 
mission and goals of the partnership, and focus on areas of commonality first in order to 
establish a level of trust between the two agencies. 

Trust is an issue for building connections between low-income African American 
communities and schools. Working through Afncan American faith-based organizations 
mitigates the trust issue, since such organizations traditionally represent the interests of African 
American people unmodified by “establishment” constraints. Once a partnership between school 
or district personnel and a faith-based organization includes true dialogue (both groups are heard, 
not just a one-way communication), the trust built can spread to others. 

A caution in looking to faith-based organizations to help educators reach parents lies in 
the fact that such agencies also struggle to reach parents. While such organizations are generally 
trusted, many parents look to them to help their children without assuming their need to be 
involved. A second danger may arise if faith-based staff see themselves as “rescuing” the youth 
from the bad influences of their families and/or communities. 

A key piece of work for AEL is leadership development among church and community 
volunteer workers. Many are comfortable working as individuals with children, but are 
unaccustomed to taking leadership roles among their peers. As long as leadership remains with 
those accustomed to leading, programs may improve in reach and effectiveness yet nothing 
essential will change in the communities themselves. 

Given the limited amount of AEL staff time and resources allocated to this project, care 
must be given to try to stay within the general framework currently established in the AEL REL 
Updated Annual Plan in terms of technical assistance services and number of faith-based sites. 

Another concern for working with faith-based organizations is the language that AEL 
staff use to communicate. Often we speak an educational lingo that others are not privy to, and 
this may unintentionally inhibit clear, complete communication/collaboration. And, for a 
program to be successful, it is important for stakeholders to take ownership from the begiiming. 



Student Records 

HOPE staff provided AEL evaluators with a copy of their students’ records via an Access 
database and a paper file. These data included student gender, race, and nine-week course grades 
for the 2001-2002 school year. For comparison purposes, data pertaining to the first and second 
nine-week grades for four core subjects (English, math, science, and history) were utilized in 
AEL’s baseline data analyses. A total of 145 of the 174 students had first and second nine-week 
grades for at least one of the above subjects and thus comprised the sample for this analysis. The 
students were fairly even by gender, with 55% males. More than half were African American 
(66%), with 34% White. Student ages ranged from 14 to 19 years (14 = 31%, 15 = 25%, 16 = 
20%, 17 = 12%, 18 = 9%, and 19 = 2%); the average age was 15.5 years (standard deviation of 
1 .4). Data on enrollment dates were not available, but a HOPE staff member did verbally 
confirm by phone that these students were enrolled during the first nine-week period. 
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The repeated measures ANOVAs for English and history found no statistically significant 
differences (main effects or interactions) in students’ grades from the first to second nine-week 
grading period. Further, there were no significant differences by gender/race groupings. 
However, for three of the four groups of students (Black males. White males, and Black females) 
both history and English grades declined slightly from the first to second nine-week period 
(history grades were unchanged for White females). 

The repeated measures ANOVA for math found a statistically significant difference 
(main effect) in students’ grades from the first to second nine-week grading period (F(l,87) = 
13.52,/? < .05), but no significant interaction between math and gender/race groupings. As well, 
significant differences were found by race and gender groupings (F(3,87) = 5.22, p < .05). 

White females scored significantly higher than all three of their counterpart groups. Overall, 
math grades declined from the first to second nine-week period for each group of students. As a 
measure of practical effectiveness, the effect sizes for these statistically significant differences 
were minimal at .13 for the difference from the first to second nine-week period and .15 for the 
race/gender grouping. In other words, the significance may be due more to statistical power 
(sample size) than actual practical meaningfulness. 

The repeated measures ANOVA for science found a statistically significant difference 
(main effect) in students’ grades from the first to second nine-week grading period (F(l,84) = 
6.50, p < .05) but no significant interaction between science and gender/race groupings. As well, 
significant differences were found by race/gender groupings (F(3,84) = 4.16,/? < .05). White 
males and White females scored significantly higher than Black males. Overall, science grades 
declined from the first to second nine-week period for each group of students. As a measure of 
practical effectiveness, the effect sizes for these statistically significant differences were minimal 
at .07 for the difference from the first to second nine-week period and .13 for the race/gender 
grouping. Again, the significance may be due more to statistical power (sample size) than actual 
practical meaningfulness. 

To show the differences in subject grades. Table 3 provides descriptive infonnation for 
the first and second nine-week grades for all students and by a breakdown of gender and race. 
Figure 1 then presents four subject-specific line graphs depicting mean scores by nine-week 
period by gender and race groupings. 

In sum, students showed the greatest variance in grades in math and science and 
generally received better grades in these two subjects than in English and history. Further, 
except for English and history for white females, students’ grades declined from the first to 
second nine-week grading period. 
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Table 3: Descriptive Statistics for HOPE Students 
Fall 2001 Subject Grades by Gender and Race 
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All 


Students 


Black Males 


White Males 


Black Females 


White Females 


Subjects 


N 


M 


SD 


N 


M 


SD 


N 


M 


SD 


N 


M 


SD 


N 


M 


SD 


1 St 9 weeks 
































English 


114 


2.1 


1.1 


37 


2.1 


1.1 


17 


2.1 


1.3 


30 


2.3 


0.9 


15 


2.1 


1.4 


Math 


110 


2.5 


1.1 


35 


2.4 


1.1 


15 


2.3 


0.7 


28 


2.4 


1.0 


16 


3.2 


0.8 


Science 


102 


2.5 


1.1 


34 


2.2 


1.2 


15 


3.0 


0.8 


24 


2.4 


0.9 


15 


2.9 


1.1 


History 


107 


2.0 


1.1 


33 


2.0 


1.0 


17 


2.4 


1.3 


26 


2.0 


1.1 


14 


2.4 


0.9 


2nd 9 weeks 
































English 


119 


2.0 


1.1 


38 


2.0 


1.0 


17 


2.0 


1.1 


34 


2.2 


1.1 


14 


2.1 


1.2 


Math 


110 


2.1 


1.2 


37 


1.6 


1.0 


17 


2.1 


0.9 


26 


2.2 


1.2 


16 


2.9 


0.9 


Science 


110 


2.1 


1.1 


35 


1.8 


1.0 


16 


2.7 


1.1 


26 


2.0 


1.0 


17 


2.6 


1.1 


History 


104 


1.9 


1.1 


36 


1.8 


1.1 


15 


2.3 


1.1 


24 


1.8 


1.0 


13 


2.5 


1.1 



Scale: 0 = F; 4 = A. 



Note: These mean scores may differ slightly from those depicted in Figure 1, since data in this 
table are based on all respondents in each group, regardless of whether or not a student had 
scores for both the first and second nine-week period for a particular subject. 




31 



Mean Grade Mean Grade 



20 




Black Male 
White Male 

3 □ 

Black Female 



White Female 



1st 9 weeks 



2nd 9 Vfoeks 



English 




Black Male 
White Male 

3 D 

Black Female 



White Female 



1st 9 weeks 



2nd 9 weeks 



Math 




Black Male 
White Male 

3 a 

Black Female 



White Female 



1st 9 wedcs 



2nd 9 weeks 




Science 



History 



Figure 1: Fall 2001 Subject Grades for HOPE Students by Gender and Race 
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AEL Skills Inventory 

Out of approximately 300, a total of 174 HOPE Youth Development Movement 
participants responded to this survey (about 58% response rate). The instrument was distributed 
in the summer of 2002 and results serve as the baseline data for this measure. 

The first two items of the AEL Skills Inventory were demographic items — gender and 
grade level. Fifty percent responded that they were female. Participants ranged from sixth grade 
to twelfth grade (4% selected postsecondary) with students most often selecting the eleventh 
grade (24%), followed by ninth grade (23%), and tenth grade (22%). 

Next, students were given a series of selected-response items that were listed under seven 
subscale headings. The subscales of the AEL Skills Inventory include Confidence, Written 
Communications, Oral Communications, Values, Decision Making, Problem Solving, and 
Planning. Students were asked to rate their feelings from 1 (Almost Never) to 5 (Almost 
Always). 

For analysis purposes, each set of items within a subscale was added together and divided 
by the total number of items to create subscale mean scores. All seven subscales had means 
ranging within a 1-point spread of 3.62 for the Written Communications subscale to 3.99 for the 
Values subscale on the 5-point scale. Standard deviations for the seven subscales were very 
similar, ranging from .75 to .81. See Table 4 for subscale descriptive statistics. 

Table 4: Descriptive Statistics for AEL Skills Inventory Subscales 



Scale: 

1 = Almost Never; 
5 = Almost Always 



Subscale 


N 


Mean 


Std. Dev. 


Confidence 


174 


3.70 


.75 


Written Communications 


174 


3.62 


.79 


Oral Communications 


174 


3.63 


.81 


Values 


174 


3.99 


.79 


Decision Making 


173 


3.80 


.75 


Problem Solving 


173 


3.70 


.81 


Planning 


172 


3.84 


.77 



One-way ANOVAs were computed comparing grade-level responses and gender 
responses on the subscale items. No statistically significant differences existed among grade 
levels, but statistically significant differences were found between male and female students in 
Written Communications (F’(l,169) = 1 .16,p < .05), Values (F’(l,169) = 10.69,/? < .05), and 
Planning (F’(l,167) = 4.16,/? < .05). For each subscale, females had significantly higher mean 
scores than their male counterparts. However, given the relatively small to moderate effect sizes 
generated (.42, .52, and .31, respectively), the magnitude of the gender differences for these tliree 
subscales may not be large enough to be practically significant, i.e., due more to statistical power 
(sample size). See Table 5 for more statistical details and Figure 2 for a visual depiction of the 
mean subscale scores by gender. 
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Table 5: Results of One-Way ANOVAs 
by Gender for AEL Skills Inventory Subscales 



22 



Subscale 


Male 


Female 


df 


F 


P 


Mean 

Diff 


ci 


Mean 


SD 


Mean 


SD 


Confidence 


3.69 


.85 


3.71 


.64 


1, 169 


0.02 


ns 


.02 


na 


Written Com. 


3.46 


.75 


3.78 


.79 


1, 169 


7.76 


.006 


.32 


.42 


Oral Com. 


3.54 


.85 


3.71 


.78 


1, 169 


1.96 


ns 


.17 


na 


Values 


3.81 


.91 


4.19 


.56 


1, 169 


10.69 


.001 


.38 


.52 


Decision Making 


3.71 


.81 


3.93 


.64 


1, 168 


3.88 


ns 


.22 


na 


Problem Solving 


3.62 


.88 


3.79 


.74 


1, 168 


1.78 


ns 


.17 


na 


Planning 


3.74 


.85 


3.97 


.62 


1, 167 


4.16 


.043 


.23 


.31 



Scale: 1 = Almost Never; 5 = Almost Always 



a> 

Ui 

O 

o 

C/D 

u 

t/5 

3 

C/D 

c 

<U 




Written Com. Values Problem Solving 



I I Female 
[T^Male 



Figure 2: Mean Scores for AEL Skills Inventory Subscales by Gender 
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Satisfaction Survey 

This survey was administered along with the AEL Skills Inventory in the summer of 
2002 to the approximately 300 high school students participating in the HOPE program and 
again in November 2002 to students as they arrived at the HOPE office to pick up their stipend 
checks. A total of 174 completed surveys were received from the first administration and 119 
from the second administration. Results are provided below comparing the two independent 
administrations. 

Students were asked what school they attended. Nearly half of the students were from 
Capital High School in the first administration (58%); Capital still had the most students in the 
second administration (36%). South Charleston High School was the next most frequently 
mentioned school, at 14% in the first administration and 21% in the second administration. See 
Table 6 for a summary of responses by administration. 

Table 6: Number and Percent of Students by Administration for Satisfaction Survey 



High School 


Administration #1 : Summer 2002 


Administration #2: Fall 2002 


Number 


Percent 


Number 


Percent 


Capital 


81 


47% 


42 


35% 


Washington 


6 


3% 


4 


3% 


Hoover 


2 


1% 


1 


1% 


Nitro 


2 


1% 


1 


1% 


Riverside 


17 


10% 


17 


14% 


Sissonville 


6 


3% 


8 


7% 


South Charleston 


19 


11% 


24 


20% 


St. Albans 


7 


4% 


7 


6% 


Other/Uni dent i fi ed 


34 


20% 


15 


13% 



Students were then given a list of HOPE services and asked to select the services they 
had received during this year. Students could select any or all of a number of services, which 
included (a) character education, (b) field trips, (c) postsecondary education awareness, (d) youth 
entrepreneurship training, (e) WIA initiative, (f) alternative sentencing, (g) transition/re-entry, 

(h) tutoring/mentoring, (i) career awareness, (j) college preparation, (k) abstinence education, 

(1) job referral, (m) intervention for Juvenile offenders, and (n) truancy diversion.* The services 
most often selected by students at both administrations were Job referral (37% and 73% for the 
first and second administrations), field trips/enrichment opportunities (32% and 75% 
respectively), and tutoring/mentoring (31% and 65%, respectively). See Figure 3 for more 
information regarding HOPE services. 



*Four activities were added after the first administration: (f) alternative sentencing, 

(g) transition/re-entry, (m) intervention for Juvenile offenders, and (n) truancy diversion. 




35 



24 




Q 1 St Admin. 
I^IFl 2nd Admin. 



Note: 2a = character education, 2b = field trips, 2c = postsecondary education awareness, 2d - youth entrepreneurship training, 2e = 
WIA initiative, 2h = tutoring/mentoring, 2i = career awareness, 2j = college preparation, 2k = abstinence education, 21 = job referral 



Figure 3; Percentage of HOPE Services Received by Students 



Students were asked to rate their level of satisfaction on a scale of 1 (Very Dissatisfied) 
to 5 (Very Satisfied) for a series of items pertaining to the HOPE services they had received. For 
the first administration, means ranged from 3.84 for the extent to which students improved 
academically to 4.1 1 for the extent to which the HOPE program positively impacted students’ 
lives; standard deviations ranged from 0.92 to 1.08. For the second administration, means were 
higher for seven of the eight items and ranged from 3.81, again for the extent to which students 
improved academically, to 4.25 for the quality of services received; standard deviations ranged 
from 0.95 to 1 .21 . See Table 7 for a summary of descriptive statistics by administration and 
Figure 4 for a visual depiction of mean item scores by administration. In general, students 
indicated greater satisfaction during the second administration (except about the extent to which 
they grew spiritually). 

One-way ANOVAs were computed comparing item responses by administration. Only 
one statistically significant difference was found between the two administrations. Students in 
the second administration rated quality of services received higher (mean 4.25, standard 
deviation 1.06) than did students in the first administration (mean 3.98, standard deviation 1.08) 
(F(l,279) = 4.34,/? < 05). As a measure of practical effectiveness, the effect size for this 
statistically significant difference was small at .25. In other words, the significance may be due 
more to statistical power (sample size) than practical meaningfulness. 
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Table 7: Descriptive Statistics by Administration for Satisfaction Survey 



Items 


Administration #1 : Summer 2002 


Administration #2: Fall 2002 


Number 


Mean 


Std. Dev. 


Number 


Mean 


Std. Dev 


3. Quality of 
services received 


163 


3.98 


1.08 


118 


4.25 


1.06 


4. Appropriateness 
for my situation 


156 


3.85 


1.02 


115 


4.05 


0.97 


5. Extent services 
met my needs 


158 


3.97 


0.92 


116 


4.19 


0.95 


6. Efficiency of 
delivered services 


158 


3.85 


1.01 


117 


4.02 


1.06 


7. Extent I ac- 
quired new skills 


155 


3.92 


0.94 


116 


4.03 


1.08 


8. Extent I improved 
academically 


150 


3.84 


1.02 


116 


3.88 


1.21 


9. Extent I grew 
spiritually 


150 


3.88 


1.00 


117 


3.81 


1.08 


10. Extent posi- 
tively impacted 
my life 


152 


4.11 


0.97 


117 


4.14 


1.05 



Scale: 1 = Very Dissatisfied; 5 = Very Satisfied 
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Figure 4: Mean Scores for Satisfaction Survey Items by Administration 
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For the following analysis, responses of satisfied or very satisfied were combined to 
indicate satisfaction; see Figure 5 for a visual depiction of item satisfaction by administration. 
For each item except the extent to which students had grown spiritually, a higher percentage of 
second-administration respondents indicated satisfaction. 




□ 1st Adm. 

I - . 1 2nd Adm. 



Figure 5: Percent of Agreement for Satisfaction Survey Items by Administration 



Students were then asked two open-ended questions. When asked what they liked most 
about HOPE, 134 students in the first administration responded with 166 discrete comments; 1 13 
students in the second administration responded with 140 discrete comments. Overall, students’ 
most-liked feature of HOPE was the caring staff (22% and 21% respectively for the first and 
second administrations). See Table 8 for a summary of the most frequent themes emerging from 
both administrations. 

When asked what they liked least about HOPE, 1 16 students in the first administration 
replied with 118 discrete comments; 102 students in the second administration provided a 
comment. More than half of the students in each group (53% and 59%, respectively) indicated 
there was nothing they disliked about HOPE. See Table 9 for a summary of the most frequent 
themes emerging from both administrations. 
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Table 8; Summary of What Students Liked Most About HOPE by Administration 



First Administration: Summer 2002 


Second Administration: Fall 2002 


Percent 


Topic 


Percent 


Topic 


22% 


Caring staff 


21% 


Caring staff 


19% 


Job experience 


20% 


Field trips/ activities 


13% 


Money 


15% 


Assistance offered 


11% 


Miscellaneous 


13% 


Job experience 


9% 


Assistance offered 


13% 


Miscellaneous 


7% 


Field trips/activities 


11% 


Money 


5% 


Tutoring 


7% 


Tutoring 


4% 


Fun/interesting 


3% 


Everything 


4% 


Something to do 






3% 


Everything 






2% 


Don’t know 







Note: Percentages may not equal 100 due to rounding. 



Table 9: Summary of What Students Liked Least About HOPE by Administration 



First Administration: Summer 2002 


Second Administration: Fall 2002 


Percent 


Topic 


Percent 


Topic 


53% 


Nothing disliked 


59% 


Nothing disliked 


14% 


Miscellaneous 


13% 


Meetings/activities 


11% 


Meetings/activities 


12% 


Miscellaneous 


7% 


Scheduling 


6% 


Scheduling 


6% 


Paperwork 


4% 


Money (amount or frequency) 


5% 


Jobs 


4% 


Jobs 


3% 


Money (amount or frequency) 


3% 


Transportation 


2% 


Tutoring 







Note: Percentages may not equal 100 due to rounding. 
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Summaries by Evaluation Question 



Question One 

What types of technical assistance are needed to help build the capacity of faith-based 
organizations to design and implement programs that support and nurture the educational 
attainment of youth, especially high-risk and at-risk youth? 

Through AEL staff reflections, a variety of technical assistance strategies to help build 
the capacity of faith-based organizations were noted. Staff also noted that training in grant 
writing helps such organizations develop programs to generate income to sustain their services. 
Similarly, sound evaluation practices are a key part of any successful grant, and AEL can 
provide expertise in this area. A third area of technical assistance is leadership development. 
According to staff reflections, many faith-based staff and/or volunteers are unaccustomed to 
serving in leadership roles; such assistance could help these workers become more comfortable 
in this capacity. 

Interviews with HOPE staff by AEL evaluators revealed perceptions of AEL’s technical 
assistance as high quality with “organizational intelligence” to understand the importance of 
research-based curriculum and activity, measurable goals and objectives, and accurate data 
collection. HOPE staff viewed AEL’s assistance as helpful and valuable, and deemed it to be a 
great asset. 

To helping other faith-based organizations design and implement programs, HOPE staff 
suggested that such organizations must first have a clear vision of what they desire to accomplish 
in terms of helping youth. They also noted that patience was critical, i.e., a focus on the long 
term because measurable improvements take time. Further, HOPE staff noted that AEL could 
provide assistance with record keeping, legal issues and/or paperwork, and identification of 
funding opportunities. 

Several suggestions from HOPE staff for improving AEL’s technical assistance services 
included continued learning about and understanding of grassroots community-level issues and 
how community organizations operate, and more involvement of AEL staff in school-site 
activities. 



Question Two 

Wliat impact does participation in an educational or social program run by a faith-based 
organization have on high-risk or at-risk youths’ overall educational experience (i.e., academic 
achievement, attendance, attitude toward learning, behavior)? 

Student interviewees told AEL evaluators of HOPE’S positive impact on their academic 
progress and their self-esteem, but inspection of academic records did not support these 
perceptions. Students noted the individualized emphasis they received, and their perceptions that 
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HOPE staff were truly interested in helping each child. In fact, the one topic most frequently 
mentioned as most liked about HOPE was the caring, committed staff 

On average, HOPE students had grades of B or C in each of the four subjects studied, 
with higher grades in math than in English, history, or science. Students’ math and science 
grades dropped significantly from the first to second nine-week period, yet effect sizes were 
minimal. And, while grades did drop from the first to second nine weeks, they were still well 
within the passing range. However, Black males consistently fared worse grade-wise for all four 
subjects for both grading periods. 

Students’ scores on the self-report AEL Skills Inventory ranged for the most part from 
3.5 to 4.0 (maximum score = 5) for each of seven behavioral areas (confidence, written 
communications, oral communications, values, decision making, problem solving, and planning), 
indicating that they felt mostly positive about actions pertaining to each area. 

In terms of satisfaction with HOPE services and the impact the program had on their lives, 
about three fourths of the students completing the Satisfaction survey indicated they were satisfied 
with the extent to which the HOPE services met their needs and positively impacted their lives. 
About two thirds indicated they were satisfied with new skills or knowledge they had gained and 
their spiritual growth. 



Question Three 

To what degree and in what ways are faith-based organizations able to engage family 
members of youth involved in their programs in activities that support the academic achievement 
of youth? 

At this point in time, HOPE staff believed that family involvement in their program was 
less than satisfactory, based on interviews conducted by AEL evaluators. HOPE staff 
interviewees mentioned that multiple siblings often participated in the program, but that parent 
involvement was lacking. They noted that activities were open to all family members, yet felt 
that most parents turned to HOPE only in a crisis situation. Further, HOPE staff also reported 
that even though parents seemed to support the program, they didn’t seem to fully understand the 
comprehensive array of services. 

Student data corroborated staffs perceptions, with few students noting active parent 
participation, according to student interviews conducted by AEL evaluators. Students thought 
parents supported their participation passively (i.e., through encouragement for attending) rather 
than actively (i.e., by becoming involved and attending activities). Finally, in interviews 
conducted by AEL evaluators, parents themselves indicated almost no active involvement in 
HOPE activities and few indicated that they actually understood the program fully. 
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CONCLUSIONS 



Several conclusions can be drawn from the findings of this evaluation. 

• The data collection instruments seem to provide a comprehensive picture of a faith-based 
organization’s youth program. The mixed-method approach of surveys, interviews, 
reflections, and academic records provides a triangulation of data that compensates for 
any inherent weaknesses in a particular collection strategy. 

• The various collection methods do not seem overly burdensome to faith-based staff or 
overly intrusive to participants. Data were collected in a timely manner, once AEL and 
HOPE staff were able to coordinate their schedules. However, there were difficulties 
encountered in identifying sufficient numbers of students and parents willing to be 
interviewed. 

• One area that seems to have been left untapped is the perceptions of faith-based staff in 
terms of their communication and collaboration with their community schools. The 
interview protocols did not include any mention of this aspect. 

• It seems that the time is ripe, with the conclusion of this report, to work closely with 
HOPE staff in building their capacity for evaluative activities, interpretations, and 
utilization. HOPE staff willingly shared all requested data and have seemed most 
appreciative of the evaluation assistance received to date. 

• It also seems apparent, given AEL staff and HOPE staff input, that AEL’s future work 
with HOPE staff could focus on capacity building in the areas of leadership development 
and training in grant writing so that HOPE staff can sustain current services. 

• On average, HOPE students earned Cs in English, history, and science, and Bs in math. 
However, students’ math and science grades dropped significantly from the first to 
second nine-week period. Further, Black males consistently fared worse grade-wise for 
all four subjects for both grading periods. 

• HOPE students report feeling positive about their actions and attitudes related to 
confidence, written communications, oral communications, values, decision making, 
problem solving, and planning. 

• HOPE students report that the HOPE services are adequately meeting their needs and 
positively impacting their lives. They show less satisfaction with knowledge or skills 
gained from their participation in the HOPE program or spiritual growth. 

• There seems to be a discrepancy regarding academic improvement as a result of HOPE 
participation. Two thirds of the students indicated they were satisfied with their 
academic improvements and interviewees noted the academic benefits of such 
participation, yet students’ grades did decline in four subject areas across grading periods. 
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RECOMMENDATIONS 



Given the conclusions noted previously, several recommendations are suggested for 
improving any future data collection efforts. 

• AEL project and evaluation staff need to work closely with faith-based staff to strategize 
ways to promote more participation (number and depth) of both students and parents in 
future group interviews. Perhaps one such strategy might be to schedule an evening 
interview, with dinner provided, so that parents and students could attend and participate 
without feeling pressured to skip or rush through the interview. 

• AEL could consider tailoring future technical assistance to HOPE staff toward training in 
grant writing and leadership, if HOPE staff are receptive to this suggestion and willing to 
work on these endeavors. 

• The interview protocol for faith-based staff should be modified to include questions 
pertaining to their perceptions of school communication/collaboration. 

• HOPE staff may want to investigate possible causes for students’ grade declines across 
grading periods. And, they may consider monitoring students’ grades more closely by 
grading periods, to try to intervene as necessary to avoid such declines. Further, they 
may want to intensify their tutoring activities with Black males, given their consistently 
lower grades. However, as Linn and Haug (2002) note, “changes in scores for . . . groups 
of students have a great deal of volatility” (p. 1). While their research pertained 
specifically to year-to-year score changes, it may be that the same principle applies to 
within-year changes, i.e., that student grades are not always a reliable measure. Linn and 
Haug offer a number of ways to improve the accuracy of results: combining data across 
multiple grades, multiple subject areas, and multiple years. While multiple grades and 
multiple subject areas are already included in the present study, the inclusion next year of 
multiple years would provide a longitudinal look at students’ scores. 
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Appendix A: Description of HOPE Programs 



HOPE YOUTH DEVELOPMENT MOVEMENT 
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Appendix B: Completed Evaluation 
Standards Checklist 



Checklist for Applying the Standards 



To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear in Joint 
Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, CA, Sage. 

The Standards were consulted and used as indicated in the table below (check as appropriate): 



Descriptor 

U 1 Stakeholder Identification 


Tlie Standard was 
addressed 

X 


The Standard was 
partially addressed 


The Standard was 
not addressed 


The Standard was 
not applicable 


U2 Evaluator Credibility 


X 








U3 Information Scope and Selection 


X 








U4 Values Identification 


X 








U5 Report Clarity 


X 








U6 Report Timeliness and Dissemination 


X 








U7 Evaluation Impact 


X 








FI Practical Procedures 


X 








F2 Political Viability 


X 








F3 Cost Effectiveness 


X 








PI Service Orientation 


X 








P2 Formal Agreements 


X 








P3 Rights of Human Subjects 


X 








P4 Human Interactions 


X 








P5 Complete and Fair Assessment 


X 








P6 Disclosure of Findings 


X 








P7 Conflict of Interest 


X 








P8 Fiscal Responsibility 


X 








A1 Program Documentation 


X 








A2 Context Analysis 


X 








A3 Described Purposes and Procedures 


X 








A4 Defensible Information Sources 


X 








A5 Valid Information 


X 








A6 Reliable Information 


X 








A7 Systematic Information 


X 








A8 Analysis of Quantitative Information 


X 








A9 Analysis of Qualitative Information 


X 








A 10 Justified Conclusions 


X 








A1 1 Impartial Reporting 


X 








A 12 Metaevaluation 


X 









The Program Evaluation Standards (1994, Sage) guided the development of this (check one): 

request for evaluation plan/design/proposal 

evaluation plan/design/proposal 

evaluation contract 

^ evaluation report 

other: — 



Name 



Kimberly S . Cowley 









Position or Title 



/f^ (signature) n 

•j, Researcn and Evaluation Specialist 



Date 



June 5 , 2003 



Agency 

Address 



AEL 

P.O. Box 1348, Charleston^ WV 25325 



Relation to Document 



Co-author 

(e.g., author of document, evaluation team leader, external auditor, internal auditor) 
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Appendix C: Interview Protocols 



AEL Group Interview Protocol 
for Faith-Based Staff 



Hello, Fm Kim Cowley from AEL. AEL is working with the HOPE program in order to 
determine how well it is achieving its goals of helping youth and to determine how other faith- 
based agencies can meet similar needs. Today Fd like to get your input regarding your 
participation in the HOPE program. 

There are a couple of “ground rules” to mention before we get started. There are no right or 
wrong answers. Everyone may not agree with each other, and that’s fine. We want each 
person’s true perceptions of their involvement; these perceptions may be positive, negative, or 
both. And, your answers are voluntary; if you don’t want to respond to a particular item, that’s 
okay. I’ll be taking notes throughout our discussion. In any written summaries, no one will be 
identified by name. 

1. How would you describe the HOPE program? (i.e., what it is and what it offers) 



2. Describe your involvement with HOPE, (i.e., length, why, current role) 



3. What are the main strengths of the HOPE program? (i.e., what works best for participants) 



4. What are the main weaknesses of HOPE program? (i.e., what do you think is least effective) 



5. How is the HOPE program different from other tutoring/mentoring programs? 
(i.e., staffing, faith aspect, focus, operations, personal growth) 



6. Describe the involvement of family members of youth involved in HOPE activities. 



7. What suggestions would you make for improving the HOPE program? 

8. What value do you place on HOPE’S involvement with AEL in impacting at-risk youth? 

9. What could AEL do to improve its services in working with the HOPE program? 

10. What type of technical assistance do you feel is needed for other faith-based organizations to 
design and implement programs that support the educational attainment of at-risk youth? 



Cowley, 10/11/02 



AEL Student Group Interview Protocol 
for HOPE Youth Development Movement 

Welcome. I am Kim Cowley from AEL. AEL is working with HOPE in order to determine how 
well it is achieving its goals of helping youth and to determine how other faith-hased agencies can 
meet similar needs. We’re wanting your input regarding your participation in the HOPE program. 

There are a couple of “ground rules” to mention. There are no right or wrong answers. Everyone 
may not agree with each other, and that’s fine. We want each person’s true perceptions of their 
involvement; these perceptions may he positive, negative, or both. And, your answers are voluntary; 
if you don’t want to respond to a particular item, that’s okay. I’ll he taking notes throughout our 
discussion, hut in any written summaries, no one will he identified hy name. 

1. Describe the types of HOPE activities in which you have participated. 

(i.e., what, when, where, how many) 

2. How did you get involved with HOPE? 

(i.e., who referred you and why) 

3. Describe how the HOPE program operates. 

(i.e., how do you receive services, what type of paperwork is required from you) 

4. What are the main strengths of the HOPE program? 

(i.e., what do you like the most, what works best) 

5. What are the benefits you are gaining from the HOPE program? 

(i.e., short- or long-term, what keeps you participating) 

6. What are the main weaknesses of HOPE program? 

(i.e., what do you like the least, what doesn’t work) 

7. How is the HOPE program different from other tutoring/mentoring programs? 

(i.e., staffing, faith aspect, focus, operations, personal growth) 

8. Describe the involvement of your family members in HOPE activities. 

9. What suggestions would you make for improving the HOPE program? 

10. Other comments you’d like to make concerning your participation in the HOPE program. 



Cowley, 5/23/02 



AEL Parent Group Interview Protocol 
for HOPE Youth Development Movement 



I am Kim Cowley from AEL, which is working with HOPE in order to determine how well it is 
achieving its goal of helping youth and to determine how other faith-based agencies can meet 
similar needs. We’re wanting your input regarding your child’s participation in the HOPE program. 

There are a couple of ground rules to mention. There are no right or wrong answers. Everyone 
may not agree with each other, and that’s fine. We want each person’s true perceptions; these 
may be positive, negative, or both. And, your answers are voluntary; if you don’t want to 
respond to a particular item, that’s okay. I’ll be taking notes throughout our discussion but no 
one will be identified by name in our written summaries. 

1 . How did your child get involved with HOPE? 

(i.e., who referred and why) 



2. Describe how the HOPE program operates. 

(i.e., how does your child [or family] receive services, what type of paperwork is required) 



3. Describe the types of HOPE activities in which your child (or family) has participated, 
(i.e., what, when, where, how many) 



4. What are the main strengths of the HOPE program? 
(i.e., what do you like the most, what works best) 



5. What are the benefits your child (or family) is gaining from the HOPE program? 
(i.e., short- or long-term, what keeps him/her participating) 



6. What are the main weaknesses of HOPE program? 
(i.e., what do you like the least, what doesn’t work) 



7. How is the HOPE program different from other tutoring/mentoring programs? 
(i.e., staffing, faith aspect, focus, operations, personal growth) 



8. What suggestions would you make for improving the HOPE program? 



Cowley, 11/14/02 



Appendix D: AEL Skills Inventory 



A EL Skills Inventory’^ 



Identification Number 
SSN Home# bay born 



The AEL Skills Inventory is composed of 70 items that focus on 
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© 


© 


© 


® 


skills that individuals may or may not possess. This is not a test 


© 


O 


© 


© 


© 


© 


and there are no right or wrong answers. Its purpose is to help 


© 


© 


© 


© 


© 


© 


you learn about your own skills. 


© 


© 


© 


© 


© 


© 




© 


0 


© 


© 


© 


© 


Please fill in your "Identification Number" to the right. This 


© 


© 


© 


© 


© 


© 


number ensures anonymity and is easy to generate. It consists 


© 


© 


© 


© 


© 


© 


of the last two digits of your Social Security Number (SSN), 


© 


© 


© 


© 


© 


© 


the last two digits of your home phone number (00 if no home 


© 


© 


© 


© 


© 


© 


phone), and the two digits for the day of the month on which 


© 


© 


© 


© 


© 


© 


you were born (01-31). 


© 


© 


© 


© 


© 


© 


When completing this survey, please fill in bubbles completely. Like this: 


• 


Not this: 






Indicate your gender and grade level. 














0 5th 0 8th 




0 


11th 




0 Female 0 6th 0 9th 




0 


12th 




0 Male 0 7th 0 10th 




0 


Postsec. 



Read each item and decide how often you possess that particular skill. Using a scale of 
1 {Almost Never) to 5 {Almost Always), fill in the bubble that best represents your choice. 



Confidence 


Almost Never 




Almost Always 


1. 


I am entitled to my own opinion. 


o 


© 


© 


© 


© 


2. 


I can state my opinion when others disagree. 


o 


© 


© 


© 


© 


3. 


I can identify my weaknesses. 


o 


© 


© 


© 


© 


4. 


I can accept constructive criticism. 


o 


© 


© 


© 


© 


5. 


I feel comfortable in most situations. 


o 


© 


© 


© 


© 


6. 


I can take a stand when the going gets rough. 


o 


© 


© 


© 


© 


7. 


I like to face my problems head on. 


o 


© 


© 


© 


© 


8. 


I can resist when other people impose on me. 


o 


© 


© 


© 


© 


9. 


I can insist that others respect my rights. 


o 


© 


© 


© 


© 


10. 


I can identify my strengths. 


o 


© 


© 


© 


© 



^Most items adapted from the Leadership Skills Inventory, Karnes A Chauvin, 1985. 
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Written Communications 



Almost Never 



Almost Always 



11. 


I can write my ideas so others understand them. 


o 


© 


© 


© 


© 


12. 


I can distinguish fact from opinion in writing. 


o 


© 


© 


© 


© 


13. 


I can compare and contrast ideas in writing. 


o 


© 


© 


© 


© 


14. 


I can summarize written information. 


o 


© 


© 


© 


© 


15. 


I know how to use written information. 


o 


© 


© 


© 


© 


16. 


I can write to persuade others to my viewpoint. 


o 


© 


© 


© 


© 


17. 


I can write a speech. 


o 


© 


© 


© 


© 


18. 


I can evaluate my writing. 


o 


© 


© 


© 


© 


19. 


I can prepare an agenda for a meeting. 


o 


© 


© 


© 


© 


20. 


I can write an outline. 


o 


© 


© 


© 


© 



Oral Communications 


Almost Never 




Almost Always 


21 


I can speak in a clear manner. 


© 


© 


© 


© 


© 


22. 


I can express a group's ideas. 


© 


© 


© 


© 


© 


23. 


I can orally defend my viewpoint. 


© 


© 


© 


© 


© 


24. 


I can deliver a prepared speech to a group. 


© 


© 


© 


© 


© 


25. 


I can moderate panel discussions. 


© 


© 


© 


© 


© 


26. 


I can use effective body language as I speak. 


© 


© 


© 


© 


© 


27. 


I am sincere when speaking. 


© 


© 


© 


© 


© 


28. 


I can deliver a speech spontaneously. 


© 


© 


© 


© 


© 


29. 


I can tell others how I feel. 


© 


© 


© 


© 


© 


30. 


I can effectively listen to others. 


© 


© 


© 


© 


© 



Values 


Almost Never 




Almost Always 


31. 


I have a set of personal standards. 


© 


© 


© 


© 


© 


32. 


I recognize the important things in my life. 


© 


© 


© 


© 


© 


33. 


I am loyal to those closest to me. 


© 


© 


© 


© 


© 


34. 


I can accept other people's ideas. 


© 


© 


© 


© 


© 


35. 


I do what I say I will. 


© 


© 


© 


© 


© 


36. 


I try to deal honestly with others. 


© 


© 


© 


© 


© 


37. 


I can accept other people's values. 


© 


© 


© 


© 


© 


38. 


I try to respect the feelings of others. 


© 


© 


© 


© 


© 


39. 


I understand my own feelings. 


© 


© 


© 


© 


© 


40. 


I treat other people fairly. 


© 


© 


© 


© 


© 



Decision Making 



Almost Never 



Almost Always 



41. 


I can gather facts for decision making. 


o 


© 


© 


© 


© 


42. 


I can analyze facts before making a decision. 


o 


© 


© 


© 


© 


43. 


I am aware of how my decisions affect others. 


o 


© 


© 


© 


© 


44. 


I know how to reach logical conclusions. 


o 


© 


© 


© 


© 


45. 


I can reach decisions on my own. 


o 


© 


© 


© 


© 


46. 


I can accept that my decisions may not be popular. 


o 


© 


© 


© 


© 


47. 


I can support group decisions, even if I don't agree. 


o 


© 


© 


© 


© 


48. 


I can make accurate decisions quickly. 


o 


© 


© 


© 


© 


49. 


I understand decision making skills. 


o 


© 


© 


© 


© 


50. 


I can accept advice from others. 


o 


© 


© 


© 


© 



Problem Solving MmostNever 




Almost Always 


51. 


I know the elements of problem solving. 


© 


© 


© 


© 


© 


52. 


I know what to do as a leader in solving problems. 


© 


© 


© 


© 


© 


53. 


I can identify a problem. 


© 


© 


© 


© 


© 


54. 


I can develop different ways to solve problems. 


© 


© 


© 


© 


© 


55. 


I can select the best way to solve a problem. 


© 


© 


© 


© 


© 


56. 


I can judge how effective my strategy is. 


© 


© 


© 


© 


© 


57. 


I can make people feel safe expressing their ideas. 


© 


© 


© 


© 


© 


58. 


I can resolve conflicts within a group. 


© 


© 


© 


© 


© 


59. 


I can work effectively for a compromise. 


© 


© 


© 


© 


© 


60. 


I can distinguish between influence and manipulation. 


© 


© 


© 


© 


© 



Planning 


Almost Never 




Almost Always 


61. 


I have organizational skills. 


© 


© 


© 


© 


© 


62. 


I set reachable goals for myself. 


© 


© 


© 


© 


© 


63. 


I seek advice when necessary. 


© 


© 


© 


© 


© 


64. 


I can tell what is needed to accomplish goals. 


© 


© 


© 


© 


© 


65. 


I can set realistic deadlines. 


© 


© 


© 


© 


© 


66. 


I can accept change. 


© 


© 


© 


© 


© 


67. 


I can meet deadlines. 


© 


© 


© 


© 


© 


68. 


I can determine whether goals are completed. 


© 


© 


© 


© 


© 


69. 


I am not overwhelmed by details. 


© 


© 


© 


© 


© 


70. 


I can set objectives to help achieve my goals. 


© 


© 


© 


© 
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Appendix E: Satisfaction Survey 



Satisfaction Survey 



The Satisfaction Survey is composed of 12 items that focus on your satisfaction with the 
HOPE Youth Development Movement. There are no right or wrong answers. Its purpose is 
to help HOPE staff understand what is working well, what might need to be changed, and 
your satisfaction with services received. 



1. What school do you attend? 



O Capital High 

O George Washington High 

O Herbert Hoover High 
O Nitro High 



O Riverside High 
O Sissonville High 
O South Charleston High 
O St. Albans High 



2. Select all of the HOPE services you have received this year. 



O Character Education/Leadership Dev. O 

O Field Trips/Enrichment Opportunities O 

O Postsecondary Education Awareness O 

O Youth Entrepreneurship Training O 

O Workforce Investment Act Initiative O 

O Alternative Sentencing for Juv. Offenders O 

O Transition/Re-entry for Juv. Offenders O 



Tutoring/Mentoring 
Career Awareness Workshops 
College Preparation 
Abstinence Education 
Job Referral 

Intervention for Juv. Off. 
Truancy Diversion 



Read each item and rate your level of satisfaction for the HOPE services you received 
this year. Using a scale of 1 ( Very Dissatisfied) to 5 ( Very Satisfied), fill in the bubble 
that best represents your choice. 



Very 0 is sat. 



Very Sat. 



3. 


The quality of HOPE services I received. 


© 


© 


© 


© 


© 


4. 


The appropriateness for my situation(s). 


© 


© 


© 


© 


© 


5. 


The extent to which the services met my needs. 


© 


© 


© 


© 


© 


6. 


The eff iciency of delivered services. 


© 


© 


© 


© 


© 


7. 


Extent to which I acquired new skills/knowledge. 


© 


© 


© 


© 


© 


8. 


The extent to which I improved academically. 


© 


© 


© 


© 


© 


9. 


The extent to which I grew spiritually. 


© 


© 


© 


© 


© 


10. 


Extent to which HOPE positively impacted my life. 


© 


© 


© 


© 


© 



11. What do you like most about HOPE? 12. What do you like least about HOPE? 
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